{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enviroment: \n",
    "Open AI gym [MountainCar-v0](https://github.com/openai/gym/wiki/MountainCar-v0)\n",
    "\n",
    "### Observation\n",
    "Type: Box(2)\n",
    "\n",
    "| Num  | Observation | Min   | Max  |\n",
    "| ---- | ----------- | ----- | ---- |\n",
    "| 0    | position    | -1.2  | 0.6  |\n",
    "| 1    | velocity    | -0.07 | 0.07 |\n",
    "\n",
    "### Actions\n",
    "\n",
    "Type: Discrete(3)\n",
    "\n",
    "| Num  | Action     |\n",
    "| ---- | ---------- |\n",
    "| 0    | push left  |\n",
    "| 1    | no push    |\n",
    "| 2    | push right |\n",
    "\n",
    "### Reward\n",
    "\n",
    "-1 for each time step, until the goal position of 0.5 is reached. As with MountainCarContinuous v0, there is no penalty for climbing the left hill, which upon reached acts as a wall.\n",
    "\n",
    "### Starting State\n",
    "\n",
    "Random position from -0.6 to -0.4 with no velocity.\n",
    "\n",
    "### Episode Termination\n",
    "\n",
    "The episode ends when you reach 0.5 position, or if 200 iterations are reached.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. gym enviroment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.40384935,  0.        ])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "env = gym.make(\"MountainCar-v0\")\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Q Table setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEARNING_RATE = 0.5\n",
    "DISCOUNT = 0.95\n",
    "EPISODES = 10000\n",
    "SHOW_EVERY = 500\n",
    "Q_TABLE_LEN = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "DISCRETE_OS_SIZE = [Q_TABLE_LEN] * len(env.observation_space.high)\n",
    "discrete_os_win_size = (env.observation_space.high - env.observation_space.low) / DISCRETE_OS_SIZE\n",
    "\n",
    "q_table = np.random.uniform(low=0, high=1,\n",
    "                            size=(DISCRETE_OS_SIZE + [env.action_space.n]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 20, 3)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_table.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decay epsilon "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon = 1  # not a constant, qoing to be decayed\n",
    "START_EPSILON_DECAYING = 1\n",
    "END_EPSILON_DECAYING = EPISODES//2\n",
    "epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Help functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_discrete_state (state):\n",
    "    discrete_state = (state - env.observation_space.low) // discrete_os_win_size\n",
    "    return tuple(discrete_state.astype(int))\n",
    "\n",
    "def take_epilon_greedy_action(state, epsilon):\n",
    "    discrete_state = get_discrete_state(state)\n",
    "    if np.random.random() < epsilon:\n",
    "        action = np.random.randint(0,env.action_space.n)\n",
    "    else:\n",
    "        action = np.argmax(q_table[discrete_state])\n",
    "    return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Rewards Recorder setup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_rewards = []\n",
    "aggr_ep_rewards = {'ep':[],'avg':[],'min':[],'max':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train the Agent "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 0\n",
      "episode: 500\n",
      "episode: 1000\n",
      "episode: 1500\n",
      "episode: 2000\n",
      "episode: 2500\n",
      "episode: 3000\n",
      "episode: 3500\n",
      "episode: 4000\n",
      "episode: 4500\n",
      "episode: 5000\n",
      "episode: 5500\n",
      "episode: 6000\n",
      "episode: 6500\n",
      "episode: 7000\n",
      "episode: 7500\n",
      "episode: 8000\n",
      "episode: 8500\n",
      "episode: 9000\n",
      "episode: 9500\n"
     ]
    }
   ],
   "source": [
    "for episode in range(EPISODES):\n",
    "    # initiate reward every episode\n",
    "    ep_reward = 0\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        print(\"episode: {}\".format(episode))\n",
    "        render = True\n",
    "    else:\n",
    "        render = False\n",
    "\n",
    "    state = env.reset()\n",
    "    done = False\n",
    "    while not done:\n",
    "        action = take_epilon_greedy_action(state, epsilon)\n",
    "\n",
    "        next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "        ep_reward += reward\n",
    "\n",
    "        # if render:\n",
    "        #     env.render()\n",
    "\n",
    "        if not done:\n",
    "\n",
    "            td_target = reward + DISCOUNT * np.max(q_table[get_discrete_state(next_state)])\n",
    "\n",
    "            q_table[get_discrete_state(state)][action] += LEARNING_RATE * (td_target - q_table[get_discrete_state(state)][action])\n",
    "\n",
    "        elif next_state[0] >= 0.5:\n",
    "            # print(\"I made it on episode: {} Reward: {}\".format(episode,reward))\n",
    "            q_table[get_discrete_state(state)][action] = 0\n",
    "\n",
    "\n",
    "        state = next_state\n",
    "\n",
    "    # Decaying is being done every episode if episode number is within decaying range\n",
    "    if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:\n",
    "        epsilon -= epsilon_decay_value\n",
    "\n",
    "    # recoard aggrated rewards on each epsoide\n",
    "    ep_rewards.append(ep_reward)\n",
    "\n",
    "    # every SHOW_EVERY calculate average rewords\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        avg_reward = sum(ep_rewards[-SHOW_EVERY:]) / len(ep_rewards[-SHOW_EVERY:])\n",
    "        aggr_ep_rewards['ep'].append(episode)\n",
    "        aggr_ep_rewards['avg'].append(avg_reward)\n",
    "        aggr_ep_rewards['min'].append(min(ep_rewards[-SHOW_EVERY:]))\n",
    "        aggr_ep_rewards['max'].append(max(ep_rewards[-SHOW_EVERY:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Rewards')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEKCAYAAADTgGjXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW5+PHPM9lDEkjCFpJAgiwJuxIQARW3CraiWFFbb2tt63ptrbfW5XbT21t/t3pva13aat1q6wIuuAvKIrgCQZEtLIEEEshGEpKQPZnv7485iRFCyDIz58zkeb9e88rMOWfOeU62Z767GGNQSimlesNldwBKKaUClyYRpZRSvaZJRCmlVK9pElFKKdVrmkSUUkr1miYRpZRSvaZJRCmlVK9pElFKKdVrmkSUUkr1WqjdAfja4MGDTVpamt1hKKVUwNi0adNhY8yQ7hwb9EkkLS2N7Oxsu8NQSqmAISL7u3usVmcppZTqNU0iSimlek2TiFJKqV4L+jaRzjQ3N1NYWEhDQ4PdofhEZGQkKSkphIWF2R2KUirI9cskUlhYSGxsLGlpaYiI3eF4lTGG8vJyCgsLSU9PtzscpVSQ65fVWQ0NDSQmJgZdAgEQERITE4O2lKWUcpZ+mUSAoEwgbYL53pRSzmJLEhGRxSKyXUTcIpLVYfsFIrJJRLZaX8/tsG+6tT1XRB4S/U+pVFBpdbdSWFPIRwc/4rmc51hftN7ukFQ32NUmsg24DHjsmO2HgYuNMYdEZBKwAki29v0VuB74DHgHmA+8659wlVLeUt1UTX5VPvnV+e1f86ryOFB9gCZ3U/txyTHJLP/2chsjVd1hSxIxxuTA8dUuxpgvOrzcDkSKSASQAMQZYz613vcscCmaRJTyCbdxU9NU06dzVDZUHpco8qvzqWioaD8mREJIjU0lLS6NuclzSYtLI21gGh8d/Igntj5BVWMVAyMG9vV2lA85uXfWt4EvjDGNIpIMFHbYV8hXJZSAdOmll1JQUEBDQwO33norra2t5OXlcf/99wPwzDPPsGnTJh5++GF+97vf8dxzz5GamsrgwYOZPn06t99+u813oILVvqp93L72dvZU7vHaOeMj4kkbmMa81HmeRGEli5TYFMJcx3dFb2xt5ImtT7CzYienJ53utTiU9/ksiYjISmB4J7t+aYx5/STvnQj8AfhG26ZODjNdvP96PFVfjBw5sss4731zOzsOVXd5TE9NGBHHby+e2OUxTz31FAkJCdTX1zNjxgxWrVrFnDlz2pPIkiVL+OUvf0l2djavvPIKX3zxBS0tLZx22mlMnz7dq/Eq1ebdvHe555N7iAiJ4Gen/YyIkIhenysmPIa0uDTSB6b3uDSRmZAJQE55jiYRh/NZEjHGnN+b94lICrAM+L4xZq+1uRBI6XBYCnCoi2s/DjwOkJWVdcJkY6eHHnqIZcuWAVBQUEBeXh6jR4/ms88+Y+zYsezatYs5c+bw5z//mUsuuYSoqCgALr74YjvDVkGqqbWJ+zfez5JdS5g2ZBoPnP0Awwd09hnQP+Ij4xk+YDg5FTm2xaC6x1HVWSIyCHgbuNsY83HbdmNMkYjUiMgsYD3wfeBhb1zzZCUGX/jggw9YuXIln376KdHR0cybN4+GhgauvPJKli5dSkZGBosWLUJEMMaROVAFkYNHD/LzD37O9vLtXDPhGm6dfmunVUz+lpGQoUkkANjVxXeRiBQCZwBvi8gKa9ctwBjg1yKy2XoMtfbdBDwB5AJ7CeBG9aqqKuLj44mOjmbnzp189tlnAFx22WW89tprvPDCC1x55ZUAzJ07lzfffJOGhgaOHj3K22+/bWfoKsh8UPABi99czIHqAzx4zoPcPuN2RyQQ8FRp5VflU9dcZ3coqgt29c5ahqfK6tjt/w389wnekw1M8nFofjF//nz+9re/MWXKFMaPH8+sWbMAiI+PZ8KECezYsYOZM2cCMGPGDBYuXMjUqVMZNWoUWVlZDByovVVU37S4W3joi4d4etvTZCZk8n9n/x+pcal2h/U1mQmZGAy7K3czbeg0u8NRJ+Co6qz+IiIignff7bwg9dZbbx237fbbb+eee+6hrq6Os846i5///Oe+DlEFsdK6Un6x9hd8Xvo5i8ct5s6Zd/apAd1XMhOtxvWKHE0iDqZJJABcf/317Nixg4aGBq655hpOO+00u0NSAWp90XruWHcH9S313Df3Pi4+xbkdNYZFDyM+Ip6dFTvtDkV1QZNIAHj++eftDkEFOLdx8/ctf+cvX/6FtLg0nrrwKU4ZdIrdYXVJRDyN6+XauO5k/XYCRqX6i8qGSm5eeTOPbH6E+WnzeeGbLzg+gbTJTMxkz5E9NLc22x2KOgEtiSgVxDaXbub2tbdT0VDBr2f9msXjFgfULM+ZCZm0uFvIPZLb3kainEVLIkoFIWMMz25/lmuXX0uoK5R/XfQvrhh/RUAlEPCMFQG0XcTBtCSiVICra65jf/X+9skO86rz2FO5h9wjuZybei6/m/s74sLj7A6zV0bGjSQ6NJqcihwWscjucFQnNIk41BtvvMGOHTu466677A5FOYDbuCmqLTpuRtz8qnxK6krajxOEpAFJpA1M48rxV3Ll+CsDrvTRkUtc2rjucJpEHGrhwoUsXLjQ7jCUTXaU72Dl/pWeRFGdz4HqAzS2NrbvjwnzTG44Y/iM9hlx0+LSGBU3isjQSBsj977MxExe3fMqre5WQlwhdoejjqFJxAb5+fnMnz+fuXPn8tlnnzF16lSuvfZafvvb31JaWspzzz3Hjh07yM7O5pFHHuEHP/gBcXFxZGdnU1xczP3338/ll19u920oH/r9Z79nW/k2UmJSSBuYxhlJZ7QnivSB6SRGJgZ0CaMnMhIyqG+pZ3/NfkYPHG13OOoYmkTevQuKt3r3nMMnw4L/6fKQ3NxcXnrpJR5//HFmzJjB888/z0cffcQbb7zBfffdx6WXXvq144uKivjoo4/YuXMnCxcu1CQS5A7VHuLSMZdy7+x77Q7Fdm3Twu8s36lJxIG0d5ZN0tPTmTx5Mi6Xi4kTJ3LeeechIkyePJn8/Pzjjr/00ktxuVxMmDCBkpKS40+ogkZzazPl9eUMix5mdyiOMHrQaMJd4f2mh1ZTaxO/+uhX3LjyxoCYxVtLIicpMfhKRMRXcxW5XK721y6Xi5aWli6PD4RfLNV7ZfVlGIwmEUuYK4wx8WPYUbHD7lB8rra5lp+t+RmfFXlm9t5duZvxCeNtjqprWhJRymHaelsNG6BJpE1mQiY7K3YG9QeoioYKfrTiR2ws3sgvsn5BiITwTt47dod1UppElHKYkloriWhJpF1mQiZVjVUU1RbZHYpPHDp6iGvevYbcI7n8+Zw/8/2J32fWiFksz1vu+MRpS3WWiCwG7gEygZnWWiGIyEysZW3xrKt+j7X2CCIyH/gzEAI8YYyxpx7KC9LS0ti2bVv762eeeabTfT/4wQ+O2w9w9OhRX4eobKQlkeN1nBZ+RMwIm6PxrtzKXG54/wbqW+t5/ILHOW2YZ5buBWkL+NXHv2LL4S1MHTLV5ihPzK6SyDbgMmBdJ9uzjDHTgPnAYyISKiIhwKPAAmAC8B0RmeDPgJXyl+LaYqJCo4gNi7U7FMcYGz8Wl7iCbtDh5tLNXLP8GgyGZ+Y/055AAM4deS7hrnDezXP2Iq62JBFjTI4xZlcn2+uMMW2typFAWzluJpBrjNlnjGkCXgQu8U+0SvlXSV0Jw6KH9ZtxIN0RFRpFelx6UPXQWle4juveu474yHj+edE/GRc/7mv7Y8NjOTPlTFbkr6DV3WpTlCfnuDYRETldRLYDW4EbraSSDBR0OKzQ2qZU0CmpK9GqrE5kJmaSUxEcJZE3977JT1f/lPSB6fxj/j9Ijun839mC9AUcrj9Mdkm2nyPsPp8lERFZKSLbOnl0WYIwxqw3xkwEZgB3i0gknvaR4w7t4trXi0i2iGSXlZX17UaU8rOS2hJtVO9ERkIGpXWllNeX2x1Knzy7/Vn+86P/JGtYFk9d+BSJUYknPPaslLOIDo12dJWWz5KIMeZ8Y8ykTh6vd/P9OUAtMAlPySO1w+4U4FAX733cGJNljMkaMmRIX25DKb9qcbdwuP6wJpFOTEj0NIMGapWWMYYHNz3IA9kPcMGoC/jL+X8hJjymy/dEhUZxzshzeH//+45dmMtR1Vkiki4iodbzUcB4IB/YCIy19ocDVwFv2BaoUj5SXl9Oq2ll+IDhdofiOG2D7gKxSqvF3cJvP/ktT257kivGXcEDZz1AeEh4t967IG0B1U3VfFr0qY+j7B1bkoiILBKRQuAM4G0RWWHtmgt8KSKbgWXAzcaYw1a7yC3ACiAHWGqM2W5H7Er5Unv3Xi2JHCcuPI6UmJSA66HV0NLAbR/cxrLcZdw09SZ+NetXPZqNePaI2cSFxzm2SsuWcSLW2I9lnWz/J/DPE7znHcD5wzeV6gMdI9K1QGtcr26q5ierfsIXpV9w98y7+W7md3t8jrCQMC4YdQHv5r1LfUs9UaFRPoi09xxVndVf5Ofnk5GRwY9//GMmTZrE1VdfzcqVK5kzZw5jx45lw4YNbNiwgdmzZ3Pqqacye/Zsdu3y9Ij+4x//yA9/+EMAtm7dyqRJk6irq7PzdpQX6Wj1rmUkZFBQU0BNU43doZxUWV0Z1y6/li2Ht/CHs/7QqwTSZn76fOpa6viw8EMvRugd/X4Cxj9s+IPXG+oyEjK4c+adXR5zsqngn332WdatW0doaCgrV67kP//zP3nllVf42c9+xrx581i2bBm///3veeyxx4iOjvZq/Mo+JXUlhLvCGRQxyO5QHKltWvhdFbvIGp5lczQnVlBTwPXvXU95QzmPnvsos5Nn9+l8M4bNIDEykXfz3uUbad/wUpTe0e+TiF3apoIHOp0KvqqqimuuuYY9e/YgIjQ3e3pmuFwunnnmGaZMmcINN9zAnDlz7LwN5WUltZ4xIjrQsHMdpz9xchK5f8P9VDVV8eQ3nmTykMl9Pl+IK4QL0y7k5d0vc7Tp6El7dflTv08iJysx+MrJpoL/9a9/zTnnnMOyZcvIz89n3rx57cfv2bOHmJgYDh06YS9nFaDaRqurzg2OGsyQqCGO7uZb11zHJ4c+4YrxV3glgbRZkL6A53c+z+qC1Sw8xTlLZ2ubiENVVVWRnOwZxdpxAsaqqipuvfVW1q1bR3l5OS+//LJNESpf0NHqJ5eRkMGOcueuLfLxoY9pcjdx7shzvXreqUOmMmLACMf10tIk4lB33HEHd999N3PmzKG19at5c2677TZuvvlmxo0bx5NPPsldd91FaWmpjZEqb3Ebt5ZEuiEzMZO8qjwaWhrsDqVTqw+sZmDEQE4deqpXzysizE+fz2eHPqOyodKr5+6Lfl+dZYfuTgW/e/fu9u2/+93vAHjqqafat6WmppKbm+vjaJW/VDRU0OJu0SRyEpkJmbSaVvZU7vFqdZE3NLubWVu4lnNSzyHU5f1/rwvSF/DUtqd4f//7XDH+Cq+fvze0JKKUQ+gYke7JSMgAnDlyfVPJJmqaarxeldVmfPx40uLSWJ6/3Cfn7w1NIko5RNsYkeHROuVJV5JjkokNj3Vk4/rqA6uJDIlk9oi+dek9ERHhovSLyC7Obv99sVu/TSJOX3KyL4L53oKZlkS6R0TITMh03PQnxhhWH1jNGSPO8Omo8vnp8zEY3tv/ns+u0RP9MolERkZSXl4elP9sjTGUl5cTGRlpdyiqh0pqSwiVUBIiE+wOxfEyEzLZXbmbZrdzZrbdUbGDkroSn1VltUkfmE5mQqZjemn1y4b1lJQUCgsLCda1RiIjI0lJSbE7DNVDJXUlDI0eikv65We7HslIzKDJ3UReVd5xKwLaZfWB1bjExdkpZ/v8WvPT5/OnTX+ioKaA1NjUk7/Bh/plEgkLCyM9Pd3uMJT6Gh0j0n0TEr5aW8RJSeS0oacRHxnv82vNT/MkkeV5y7luynU+v15X9COPUg6hKxp236i4UUSFRjmmXeRA9QFyj+T6vCqrzYiYEUwbMo138+2v0tIkopQDGGN0oGEPhLhCGBs/1jHdfNcUrAHgnNRz/HbN+enz2VO5h9xKe8eKaRJRygGqGqtobG3U6qweyEzIZFfFLtzGbXcorD6wmvHx40mJ9V9b5IVpF+ISl+2lEbtWNlwsIttFxC0ix03FKSIjReSoiNzeYdt8EdklIrkicpd/I1bKt3RFw57LTMjkaPNRCmsKbY2jvL6cL0q/8FtVVpvBUYOZMXwGy/OW29rT1K6SyDbgMmDdCfb/CWhPryISAjwKLAAmAN8RkQm+DlIpf9ExIj3XcVp4O60tXIvB+D2JAFyUfhEHag7YOiGlLUnEGJNjjNnV2T4RuRTYB3RcQ30mkGuM2WeMaQJeBC7xfaRK+UdxbTGgJZGeGDNoDKESanvj+uoDqxkxYATj48f7/drnjTyPUFeorWNGHNUmIiIDgDuBe4/ZlQwUdHhdaG070XmuF5FsEckO1rEgKriU1JXgEheDowbbHUrACA8JZ0z8GFunP6lrruPTQ59y7shzbVlIbGDEQOaOmMvy/OW2tQ35LImIyEoR2dbJo6sSxL3An4wxR489XSfHnrAS0BjzuDEmyxiTNWTIkN6Er5RfldSWMDhqsE9mfg1mGQkZ5FTk2NYm4Ku1Q3pifvp8SupK+KL0C1uu77PfWGPM+b142+nA5SJyPzAIcItIA7AJ6DgsMwXQZf1U0CipK9GJF3shIyGD13Jfo7Su1Jb2JF+tHdIT56SeQ2RIJO/mvcv0YdP9fn1HVWcZY840xqQZY9KAB4H7jDGPABuBsSKSLiLhwFXAGzaGqpRX6Wj13pmQ+NXIdX9rWzvk7JSzbS1BRodFc3bq2by//31a3C1+v75dXXwXiUghcAbwtois6Op4Y0wLcAuwAsgBlhpjtnf1HqUChTGG4tpibVTvhfHx4xGEHRX+753k67VDemJB2gIqGirYULTB79e2JX0aY5YBy05yzD3HvH4HeMeHYSlli6PNR6lvqdck0gvRYdGMihvFznL/l0R8vXZIT8xNmUtMWAzv5L3D7GT/xuOo6iyl+qO2xYW0Oqt3MhMy/T5WpG3tkNkjZvt07ZDuigiJ4NyR57LqwCqaWpv8em1NIkrZTEer901mYiZFtUUcaTjit2v6a+2Qnrgo/SKONh/lw4Mf+vW6mkSUspmOVu+btjXXd1b6r0rLn2uHdNfMpJnER8SzPM+/669rElHKZm3VWUOjhtocSWDKTLCmP/HjyPXVB1Yzfdh0BkUO8ts1TybMFcY30r7B2sK11DXX+e26mkSUsllJXQmJkYmEhYTZHUpAGhQ5iKQBSX5rF2lfOyTVOVVZbeanzae+pZ4PCj7w2zU1iShls+K6Yq3K6qOMhAy/lUTa1w4Z6b+1Q7rrtGGnMTR6qF+nh9ckopTNdEXDvstMzGR/9X6/VOOsPrCajIQMkmNOOH2fbVziYn7afD46+BFVjVX+uaZfrqKUOiFd0bDvMhMyMRh2V+726XXa1w5xYFVWmwXpC2hxt7D6wGq/XE+TiFI2qmuuo6apRquz+qitcd3X62rYuXZId01MnEhqbCrv5PlnbLYmEaVspGNEvGNo9FASIhN8PofWqgOrSI5JZlz8OJ9epy9EhPlp86lpqqHZ3ezz62kSUcpGbUlk+ACdwbcvRKR9WnhfqW2u5bNDn3FO6jm2rB3SEzdPu5kXv/UiYS7f9/jTJKKUjdqnPNGSSJ9lJmSSeyTXZ9N+fHzQ/rVDusufswprElHKRm0lkaHROtCwrzISM2hxt5B7JNcn519dsJpBEYNsXTvEiTSJKGWjktoSBkUMIjI00u5QAt6EBN+tLdLsbmZdwTrb1w5xIk0iStlIu/d6T0psCgPCBvhk0GF2cTY1zc5YO8Rp7FqUarGIbBcRt4hkddieJiL1IrLZevytw77pIrJVRHJF5CFxesuWUt2gKxp6j0tcjI8f75PG9ba1Q84YcYbXzx3o7CqJbAMuA9Z1sm+vMWaa9bixw/a/AtcDY63HfN+HqZRv6Wh175qQOIHdlbtpdbd67ZzGGFYXOGftEKexJYkYY3KMMbu6e7yIJAFxxphPjTEGeBa41GcBKuUHja2NVDZWahLxooyEDOpb6tlfvd9r59xRvoPSulKtyjoBJ7aJpIvIFyKyVkTOtLYlA4Udjim0tikVsEprSwFdR8SbMhOtaeG9WKW16sAqx60d4iQ+SyIislJEtnXyuKSLtxUBI40xpwL/ATwvInFAZ+0fpotrXy8i2SKSXVZW1rcbUcpHiuuKAR1o6E3pA9MJd4V7tXF9TcEax60d4iQ+66tmjDm/F+9pBBqt55tEZC8wDk/JI6XDoSnAoS7O8zjwOEBWVtYJk41SdtIpT7wvzBXGuPhxXuvmu796P7lHcrlzxp1eOV8wclR1logMEZEQ6/loPA3o+4wxRUCNiMyyemV9H3jdxlCV6jMdre4bGYme6U88zad9s+aAc9cOcQq7uvguEpFC4AzgbRFZYe06C9giIl8CLwM3GmMqrH03AU8AucBewH+rrijlAyV1JcSGxxIdFm13KF63/VAVOw5V23LtzIRMqpuqOVR7wsqKbltd4Ny1Q5yiW9VZInIKUGiMaRSRecAU4FljzJHeXNQYswxY1sn2V4BXTvCebGBSb66nlBMFY/deYwxPfZzPfe/kMHrwAN7/D/83RrdNC7+zfGef/vkfrj/M5tLN3DT1Jm+FFpS6WxJ5BWgVkTHAk0A68LzPolKqHwi2gYZ1TS3c+uJmfvfWDuIiQ9lbdpSGZu+N1+iusfFjCZEQdlT0bW2RtQXOXzvECbqbRNzGmBZgEfCgMeY2IMl3YSkV/ErqShgeHRw9s/aX13LZXz7hzS2H+MWF4/n9osm4DewuqfF7LJGhkaQPTGdd4Trey3+P3ZW7aWxt7PF5VhesdvzaIU7Q3d5ZzSLyHeAa4GJrm+8nqlcqSDW3NlNeXx4U1VlrdpZy64tf4HIJ/7h2JmeNG0L+4VoAdhbVMCXF/11jz0k9h79v/Ts/X/tzAARhRMwI0uLSSBuY9rWvw6KHHbc+SNvaIVeMv8Lxa4fYrbtJ5FrgRuD3xpg8EUkH/uW7sJQKbmX1ZRhMQFdnud2Gh1fn8uCq3WQOj+Ox700nNcHTSWBkQjRRYSHsKLKncf2np/2UH0/+Mfur95NfnU9+VT551XnkV+Xzeenn1LfUtx8bFRrlSSodEktpXWnArB1it24lEWPMDuCnHV7nAf/jq6CUCnaBPkakqr6Z/1iymVU7S7ns1GTuu2wykWEh7ftdLmH88Fh2FtuTRACiw6LJTMxsH8XexhhDaV1pe3LJr/YkmC2Ht7A8fznGGsesa4d0T5dJRES20sXIcGPMFK9HpFQ/EMhjRHYV13DDP7MprKznvy6ZyPdmjeq0yiczKY53thZhjHFUlZCIMGzAMIYNGMbpSad/bV9jayMHqg+QX51P0oAkXTukG072HfqW9fXfra//tL5eDdT5JCKl+oH2kkiAVWe9+eUh7nh5CzGRobx4/Syy0hJOeGxmUiwvbDhAcXUDSQMDY/bbiJAIxsaPZWz8WLtDCRhdJhFjzH4AEZljjJnTYdddIvIx8F++DE6pYFVcW0x0aDQxYTF2h9ItLa1u/rB8J3//MI+sUfH85erTGBrX9WqMmUlxAOQUVQdMElE9190uvgNEZG7bCxGZDQzwTUhKBb+2MSJOquY5kcNHG/m3J9fz9w/zuOaMUTx/3ayTJhCA8cNjAcgp8n83X+U/3a3w+yHwtIgMxNNGUmVtU0r1QqAsi7u54Ag3/WsTFbVN/N/iqXx7esrJ32SJiwwjJT6KncWaRILZSZOIiLiAMcaYqW3TshtjqnwfmlLBq6S2hFlJs+wOo0svbjjAb17fztC4CF65aTaTkgf2+BwZw+PIsambr/KPk1ZnGWPcwC3W82pNIEr1TYu7hcP1hx3dqP7Qqj3c9epWTh+dwJu3zO1VAgFP4/o+m6Y/Uf7R3TaR90XkdhFJFZGEtodPI1MqSJXXl9NqWh1bnWWM4YUNBzhz7GCeuXYm8QPCe32uzKQ43Ab2lBz1YoTKSXrSJgJfdfUFT9vIaO+Go1Twa+ve69QVDQsr6ymqauCmeacQ4upbw39GW+N6cTWTU3pXmlHO1t0R6+m+DkSp/sLpo9U35HmW8JnRxRiQ7hqVOICosBB2ag+toNXt4ZgiMgmYALT37TPGPOuLoJQKZk4frb4xv4K4yFDGD4vt87lCXMK44bHauB7EutUmIiK/BR62HucA9wMLe3tREVksIttFxC0iWcfsmyIin1r7t4pIpLV9uvU6V0QekkDoYK9UJ0rqSogIiWBghDOrdzbkVzAjLQFXH6uy2kxI8syh5Y3lapXzdLdh/XLgPKDYGHMtMBWI6MN1twGXAes6bhSRUDyzA99ojJkIzAOard1/Ba7Hs+76WGB+H66vlG3aVjR04uegsppG9pXVMiPde/1mMobHUVnXTEl1z9f0UM7X3SRSb3X1bbHGipTSh0Z1Y0yOMWZXJ7u+AWwxxnxpHVdujGkVkSQgzhjzqfF8nHkWuLS311fKTk5e0TA739MeMtOrSeSrxnUVfLqbRLJFZBDwd2AT8DmwwQfxjAOMiKwQkc9F5A5rezJQ2OG4Qmtbp0TkehHJFpHssrIyH4SpVO85ebT6+rwKIsNcTBrhvaq2jA5zaKng093eWTdbT/8mIsvxlAq2dPUeEVkJdNaH8ZfGmNe7iGcuMAPPLMGrRGQT0NlvX1dT1D8OPA6QlZWlFbHKMdzG7egksjG/gtNGxhMe2t3Plyc3MCqM5EFR2kMrSHUriYjIs8CHwIfGmJ3deY8x5vxexFMIrDXGHLau+w5wGp52ko6T9qQAh3pxfqVsVdFQQYu7xZHVWdUNzewoquan53p/GvTMJHsXqFK+092PG88AScDDIrJXRF4RkVt9EM8KYIqIRFuN7GcDO4wxRUCNiMyyemV9HzhRaUYpx3LyGJFN+ysxxrvtIW0yhsext6xWpz8JQt1KIsaY1cDvgV8DTwBZwE29vaiILBKRQuAM4G0RWWH4tVdqAAAgAElEQVRdpxL4I7AR2Ax8box523rbTda1c4G9wLu9vb5SdmkfI+LAksjGvApCXcKpIwd5/dyZSXG0ug25pTr9SbDpbnXWKjzrh3yKp1prhjGmtLcXNcYsA5adYN+/8FRfHbs9G5jU22sq5QROLolsyKtgUvJAosO9vyRsRlLb2iLVvZ7MUTlTd6uztgBNeP6JTwEmiYguVaZUD5XUlhDqCiUh0lnzlzY0t7KlsIrTfVCVBZCWOIDIMJeuLRKEuts76zYAEYkBrgWextPzqi8DDpXqd9p6ZrnEe72fvGFzwRGaWt1emS+rMyEuYfwwnf4kGHW3OusW4ExgOrAfeApPtZZSqgec2r13Y14FIt6ZdPFEMobH8d6OYowxjhytr3qnu5WfUXgavDcZY1p8GI9SQa2ktoSJiRPtDuM4G/IrGD8sloHRYT67RmZSLEuyCyiraezWGu0qMHS3d9YDQBjwPQARGSIiOj28Uj1gjHHklCctrW4+31/p01IIfDVyfYdWaQWVnszieydwt7UpjE56UCmlTqyqsYrG1kbHVWftKKqmtqnVJ+NDOsoc7kki2rgeXLrburcIz9TvtQDGmENA3xcbUKofae/e67CSSNsiVL5OIgOjwxgxMFIb14NMd5NIkzV7rgEQkQG+C0mp4FRcWww4b4zIhrwKRiVGM8wP7RQZSXE6h1aQ6W4SWSoijwGDROQ6YCWe0eNKqW5y4kBDt9uw0VqEyh8yk2LZW3aUxhad/iRYdHecyP+KyAV4ZtMdD/zGGPO+TyNTKsgU1xYTIiEMjhpsdyjt9pYdpbKumZl+SiIZw+NosaY/mejF6eaVfbo9v4GVNN4HEJEQEbnaGPOczyJTKsiU1JUwOGowIa4Qu0Npt95P7SFtMq0eWjuLajSJBIkuq7NEJE5E7haRR0TkG+JxC7APuMI/ISoVHJzYvXdjfgVDYiMYlRjtl+ulJUYTEerSxvUgcrKSyD+BSjwTL/4Y+AUQDlxijNns49iUCioltSWMjff+Wh29ZYxhQ14FM9MT/DaCPDTExfjhsdrNN4icLImMNsZMBhCRJ4DDwEhjjP4GKNUDbQMN5ybPtTuUdoWV9RRVNfitPaRNxvBYVuWU6vQnQeJkvbOa254YY1qBPE0gSvVcTXMN9S31DB/Q2YrR9tiY79/2kDYZw+Mor22i7GijX6+rfONkSWSqiFRbjxo8qw5Wi0iNiPS6UlNEFovIdhFxi0hWh+1Xi8jmDg+3iEyz9k0Xka0ikisiD4l+hFEBpH0xKgd1792QV0FcZCjjh/l33HDHxnUV+LpMIsaYEGNMnPWINcaEdnge14frbgMuA9Ydc73njDHTjDHT8MzTld+h7eWvwPXAWOsxvw/XV8qvnDhafUN+BVlpCbhc/v08ltlhgSoV+GxZ1MAYk2OM2XWSw74DvAAgIklAnDHmU2vk/LPApT4OUymvcVpJpKymkX1ltX6vygIYFB1O0sBIbVwPEt5fB9N7rgQusZ4nA4Ud9hVa25QKCCV1JQjCkKghdocCQLbVHuKvkerHyhiuC1QFC58lERFZiWf1w2P90hjz+kneezpQZ4zZ1rapk8NMF++/Hk/VFyNHjuxewEr5UEldCYlRiYSF+G69jp5Yn1dBZJiLyTatd56ZFMeHew7T1OImPNRZqzyqnvFZEjHGnN+Ht1+FVZVlKQRSOrxOAQ51ce3HgccBsrKyTphslPKXklpnrWi4Mb+CU1PjbfsHnpH01fQnE0b0pXlV2c1xHwFExAUsBl5s22aMKQJqRGSW1Svr+0CXpRmlnMRJy+JWNzSTU1RtS3tIm8zhnsb1ncVapRXobEkiIrJIRAqBM4C3RWRFh91nAYXGmH3HvO0mPDMH5wJ7gXf9EqxSXlBS65wpTzbtr8Rt/D8+pKP0wQMID3Vp43oQsKVh3RizDFh2gn0fALM62Z4NTPJtZEp5X21zLTXNNY4piWzMqyDUJZw6cpBtMYSGuBg3LEYb14OA46qzlAo2ThsjsiGvgknJA4kOt7dzZubwOHJ0wGHA0ySilI85aYxIQ3MrWwqrbK3KapORFMfho42U1ej0J4HMyeNElAoKbSWR4dH2z5u1ueAITa1uv0+62Jm2kes7i6sZEuuM8TNtjtQ1sXZ3GQ3NrdQ3tVLf7Ka+uZWG5lbqmlqob3J79rXvb/3666ZWTh0Vz7M/nGn3rficJhGlfKytJDJ0wFCbI/G0hwBkpcXbHIlnIkbwTH9y5ljnJJGWVjff+fv6TttrosJCiAoPISoshMgwF1HhIUSHhRIXFcawuIj2/SXVjazeWcq2g1VMsmksjr9oElHKx0rqSoiPiCciJMLuUNiQX0HG8FgGRYfbHQoJA8IZFhfhuIkYn99wgJyiav7fZZM5a9wQosJCiA4PISLU1e2p66vqmplx30peyi4I+iSibSJK+ZhTVjRsaXXz+f5K26Y66UxmUhw5DurmW360kf9dsYs5YxK5akYqyYOiSBgQTmRYSI/WPhkYHcaFE4fz2uZDNDS3+jBi+2kSUcrHnDJafUdRNbVNrcxwQKN6m4zhceSW1tDU4rY7FAAeWLGLuqZW7rl4Yp8XzLoiK4Wq+mZW5pR4KTpn0iSilI85ZbT6Bqs9xAmN6m0yk2JpbjXsO3zU7lDYXHCEJdkFXDsnjbFeWGNl9imDGTEwkqXZhSc/OIBpElHKhxpaGjjSeMQR1Vkb8ioYmRDN8IGRdofSrm2BKrsHHbrdht++vo3BMRH89LyxXjlniEu4fHoKH+4p49CReq+c04k0iSjlQ6V1pYD9Y0TcbsPG/ApHjA/paPTgAYSHuGxvXF+aXcCXhVX88qJMYiO9N9Py5dNTMQZe2RS8pRFNIkr5kFNGq+8tO0plXbOjqrLAM/3J2GExtjauH6lr4g/LdzIjLZ5Lpo3w6rlHJkYza3QCL20qxO0OzgnFNYko5UPFtcWA/SWR9W3tIQ4riYDVQ8vG6qw/vr+bqvpm7l04qc+N6Z25IiuVAxV1bLAWAgs2mkSU8qH2kojNSWRjfgVDYiMYlRhtaxydyRgeS1lNI4eP+n/6k+2HqvjXZ/v53qxRPlvXZMGkJGIiQlmaXeCT89tNk4hSPlRSW0JseCzRYfb98zbGsCGvgplpCT75pN1XbY3r/m4XMcbw29e3Ex8dzn9cMN5n14kKD+HiqSN4d2sxNQ3NPruOXTSJKOVDTujeW1hZT1FVgyOrssBTEgH/L1D12uaDZO+v5M75GQyM9u2yxYuzUqhvbuXtLUU+vY4d7FqUarGIbBcRt4hkddgeJiL/EJGtIpIjInd32DdfRHaJSK6I3GVH3Er1lBNGq2+06uKdNFK9o8SYCIbGRrDDj+0iNQ3N3PfOTqamDuLy6Sknf0MfnZo6iDFDY4KySsuuksg24DJg3THbFwMRxpjJwHTgBhFJE5EQ4FFgATAB+I6ITPBnwEr1Rkltie2z927IqyAuMpTxw/s+gM5XMpPi/Fqd9eeVezh8tJH/WjgRl8v3VXwiwhVZKXx+4Ai5pfYPrPQmW5KIMSbHGLOrs13AABEJBaKAJqAamAnkGmP2GWOa8Ky/fonfAlaqF5pbmylvKLe9OmtDfgVZaQmE+OGfZW9lJMWSW3qU5lbfT3+yu6SGpz/J56oZqUxN9d/qjpeemkyIS3hpU3CVRpzWJvIyUAsUAQeA/zXGVADJQMfvfKG1TSnHKq23BhraWJ1VVtPIvrJax7aHtJmQFEdTq5t9ZbU+vY4xhnve2E5MRCi/uDDDp9c61tDYSM4ZP5RXPz9Iix+Spb/4LImIyEoR2dbJo6sSxEygFRgBpAM/F5HRQGcfoU44ckdErheRbBHJLisr69N9KNVbTljRMNvh7SFt2tYW8XXj+jtbi/lkbzm3f2McCQP8Px3+FVkplNU0snZ38Pxf8lkSMcacb4yZ1Mnj9S7e9l1guTGm2RhTCnwMZOEpeaR2OC4FONTFtR83xmQZY7KGDHHOYjeqf3HCGJEN+RVEhrmY7PA1LUYP8Ux/4svG9bqmFv777R1MSIrju6eP8tl1unJOxlAGx4QHVQO706qzDgDniscAYBawE9gIjBWRdBEJB64C3rAxTqVOqr0kYmN11oa8Ck5NjSc81Gl/6l8XFuJizNAYnzauP7I6l6KqBv7rkom2tQ+FhbhYdGoyq3JKbRlc6Qt2dfFdJCKFwBnA2yKywtr1KBCDp/fWRuBpY8wWY0wLcAuwAsgBlhpjttsQulLdVlJXQnRoNDFhMbZcv7qhmZyiaketH9KVjKRYn01/sq/sKH//cB+XnZZMls1Ve4uzUmlxG1774qCtcXiLLcvjGmOWAcs62X4UTzffzt7zDvCOj0NTymvaxojYNUp80/5K3AZOD5AkMiEpjlc/P0j50UYSY7y3lLAxhnvf3EFEaAh3LfBvY3pnxg2LZWrqIJZmF/CjuemOnEWgJ5xdxlUqgNm9ouHGvApCXcKpI/3XjbUv2hrXd3l5Rt+VOaWs3V3Gz84fy9BYZ6ylckVWCrtLjrKlsMruUPpMk4hSPlJcV2xvo3peBROTBxIdbkuFQ49lJnkGQ3qzcb2huZX/ems744bFcM3sNK+dt68unjqCiFBXUIwZ0SSilA+0uFs4XH/Ytkb1huZWthRWBUxVFnimPxkSG8FOL5ZEHlu7j4KKeu5ZOJGwEOf8u4uLDOOiyUm8vvkQDc2tdofTJ875rioVRA7XH8Zt3LaVRDYXHKGp1e348SHHyhjuvcb1goo6/vJBLt+aksTsUwZ75ZzetHh6CjUNLazYXmx3KH2iSUSpTty3/j5+tOJHPLv9WQqqe17l0DZGZPgAe+bN2pjXNsgw3pbr99aEpDj2lBz1yoju3721A5cIv/xmphci875ZoxNJiY/ipezAXjo3MCpLlfKjgpoCXtz5IoMiBrGheAMPZD/AKQNPYV7qPOalzmPy4MmEuEK6PIfdo9XX7Slj/LBYBkX7f1R2X2QkxXqmPzlcy7hhvZ8w8v0dJby3o4Q75o8naWCUFyP0HpdLWDw9lQdX7aagoo7UBOctGNYdmkSUOsZLu1/CJS5euvglmt3NrC1cy5qCNfxj+z94ctuTJEQmcHbK2cxLncespFmdLjhl52j1jfkVbMyv5FcO/QTelbYFqnKKqnuVRHJLa/jT+3t4e2sRY4bG8KO56d4O0au+PT2ZB1ft5pXPC/nZ+ePsDqdXNIko1UFTaxOv7XmNeanz2hvFr868mqszr6a6qZqPD37MmoI1rNy/kmW5y4gIiWBW0izmpc7j7JSzGRLtmWanpLaEiJAIBkb4f7qRh1btYXBMOFfbNLVHX4weHENYiLCzuKZH03TvL6/lzyv38Nrmg0SFhfCTc8fw4zNHExHadYnRbinx0cw5ZTAvZRfy03PH+mVaem/TJKJUB+/tf4/KxkquGH/FcfviwuNYkL6ABekLaHY383nJ53xQ8AFrCtawtnAtAJMHT2Ze6jxyKnIYFu3/gYafH6jkwz2HuXtBBlHhzv4H2pnwUBdjhna/cf3gkXoeWb2HpdmFhIUI1505mhvOPsWWyRV7a3FWCre+uJnP9pUze4zzOgCcjCYRpTpYumspI2NHMitpVpfHhbnCOD3pdE5POp07ZtxB7pFcPij4gA8KPuDhLx4GYObwmf4I+WseWrWH+Ogw/m1W4JVC2mQOj+WTveVdHlNa3cBfPtjL8+sPAPC9WaO4ed4pDI1zxmDCnrhw4nBiI0NZml2gSUSpQLa7cjdflH7B7Vm345Lud1wUEcbGj2Vs/Fium3IdZXVlfHTwI8YnjPdhtMf7suAIH+wq4xcXjmdAROD+aWckxfLqFweprG0i/pgSRUVtE4+t3cs/Ps2nudVwRVYKt5w7luRBzmw8747IsBAumTaCl7ILube+mYFRvl3v3dsC9zdNKS9bumsp4a5wLjmlb4tmDokewqKxi7wUVfc9vHoPg6LDHDUyuzfaG9eLq9vHd1TVN/Pkh/t48qM86ppbWTQtmZ+eN5a0wQPsDNVrrshK5V+fHeCtLYcCri1Lk4hSQG1zLW/ufZP56fMZFBkYc011tO1gFStzSvn5BeOICeBSCHRYoKqohqkpg3jmk3weW7uX6oYWvjk5iZ+dP5axfej+60STkweSMTyWpdmFmkSUCkRv73ubupa6ThvUA8HDq/cQGxnKNXPS7A6lz4bERjA4JoIXNx7gkTW5VNQ2cX7mUG67YBwTRzh7ca3eEhEun57Cf7+dw+6Smj6NkfE3HbGu+j1jDEt2LSEjIYMpg6fYHU6P5RRVs2J7CT+ck05cZGDVp5/IpOQ4dpccZeKIOJbdPJsnrpkRtAmkzaJTkwl1CS8F2KqHWhJR/d6XZV+yu3I3vznjNwG5tsPDq/cQGxHKD+c4e2BdT/x+0WTKjzYyJSXwqhZ7KzEmgvMzh/Hq5we5Y36GoyaM7IpdKxsuFpHtIuIWkawO28NF5GkR2SoiX4rIvA77plvbc0XkIQnEv3blSEt2LSEmLIZvpn/T7lB6bFdxDe9sLeYHc9IYGB0cpRCA5EFR/SqBtFmclUJ5bROrd5baHUq32ZXqtgGXAeuO2X4dgDFmMnAB8H8i7X0t/wpcD4y1HvP9E6oKZpUNlazIX8HFp1zc6fQlTvfImlwGhIcEVSmkPzt73BCGxEYE1KSMtiQRY0yOMWZXJ7smAKusY0qBI0CWiCQBccaYT40xBngWuNRvAaug9VruazS7m7liXOA1qOeWHuWtLYf4/uy048ZTqMAUGuLi26elsGZXKaU1DXaH0y1Oq3T7ErhEREJFJB2YDqQCyUDH1FxobVOq19zGzUu7X2L6sOmMiR9jdzg99uiaXCJDQ/ixwycZVD2zOCuFVrdh2ecH7Q6lW3yWRERkpYhs6+TR1Uiup/AkiGzgQeAToAXorP3DdHHt60UkW0Syy8rK+nIbKoh9euhTCmoKuHL8lXaH0mN5h2t5ffNBvnfGKBJjIuwOR3nRKUNimD4qnqc/zmdPiXfXm/cFnyURY8z5xphJnTxe7+I9LcaY24wx04wxlwCDgD14EktKh0NTgENdnOdxY0yWMSZryJAh3rolFWSW7FpCQmQC54883+5QeuyR1bmEh7q47szRdoeifOA335pAi9vNpY9+zPJtRXaH0yVHVWeJSLSIDLCeXwC0GGN2GGOKgBoRmWX1yvo+cMJkpNTJFNcWs7ZwLZeNvYywkMDq1bS/vJbXNh/k6tNHMSRWSyHBaGrqIN78yVzGDIvlxn99zgMrdtLqPmHli63s6uK7SEQKgTOAt0VkhbVrKPC5iOQAdwLf6/C2m4AngFxgL/CuH0NWQebl3S9jjOHycZfbHUqP/WXNXkJcwg1naSkkmCUNjGLpDbO4akYqj67Zyw+f2UhVXbPdYR3HlsGGxphlwLJOtucDnU59aozJBib5NjLVHzS7m3l1z6ucmXImyTGB1T+joKKOVz4v5N9mjQrIac9Vz0SEhvA/357ClJRB/PaNbSx89CMe+9709vnFnMBR1VlK+cOaA2soqy8LyAb1v67di0uEG87WUkh/8t3TR/Li9WdQ39TKokc/4a0tJ2wS9jtNIqrfWbprKSMGjGDOiDl2h9Ijh47U81J2AVfOSCVpYOCun6F6Z/qoeN76yVwmjIjjlue/4P+9k0NLq9vusDSJqP5lX9U+1hevZ/H4xYS4Amv52L9+sBeAG+edYnMkyi5D4yJ54bpZfG/WKB5bt49rnt5ARW2TrTFpElH9yku7XiLUFcqiMf5fNKoviqsaWLKxgMunpwb0Kn6q78JDXfzu0kncf/kUNuZXcvHDH7HtYJVt8WgSUf1GfUs9r+99nQtGXkBiVKLd4fTI39buxW0MN2spRFmuyErlpRvOwG0M3/7rJyz7wp75tjSJqH5jed5yappqAm7hqdLqBl7YcIDLTksmNSHwJolUvtM2nmRa6iBuW/Il9765nWY/t5NoElH9xpJdSxgzaAzTh023O5QeeXzdPlrchn8/J/Dm91K+Nzgmgn/9+HR+OCedpz/O59+eWM/ho41+u74mEdUvbD+8ne3l21k8bnFALTx1+Ggj/1q/n0unJTMqcYDd4SiHCgtx8ZuLJ/DgldP4svAIFz/8EV8WHPHLtTWJqH5hya4lRIVGcfEpF9sdSo/8fd0+mlrc/Ps52haiTu7SU5N5+cbZuET4wdMbqG1s8fk1dXlcFfSqGqt4N+9dvjn6m8SGx9odTreVH23k2U/3s3DqCEYPibE7HBUgJiUP5K2fzCWnuJoBEb7/F68lERX03tz7Jg2tDQE3Qv3Jj/JoaGnllnO1LUT1TPyAcGafMtgv19IkooKaMYalu5cyZfAUMhMz7Q6n247UNfGPT/L51pQRjBkaOKUn1f9oElFBbWPxRvKq8gKqW29pTQO/fzuH2qZWfqKlEOVw2iaigtqSXUuIC4/jwrQL7Q6lS1X1zazYVswbXx7ik72HcRu4+vSRjBumpRDlbJpEVNAqqytj9YHVfDfzu0SGOm/a9IbmVlbllPL65oN8sKuMplY3aYnR3HLOGBZO02osFRhsSSIi8gBwMdCEZ4Gpa40xR6x9dwM/AlqBnxpjVljb5wN/BkKAJ4wx/2NH7CpwvLrnVVpMi6Oqsppb3Xyce5g3Nh9ixfZiaptaGRobwffOGMXCqSOYkjIwoMaxKGVXSeR94G5jTIuI/AG4G7hTRCYAVwETgRHAShEZZ73nUeACPOutbxSRN4wxO2yIXQWAVncrL+95mVlJsxgVN8rWWNxuw6YDlbyx+RBvby2ioraJuMhQLp46goVTR3D66ERCXJo4VGCya2XD9zq8/AxoW6P0EuBFY0wjkCciucBMa1+uMWYfgIi8aB2rSUR1al3hOopri7lrxl22XN8YQ05RDa9/eZC3vizi4JF6IsNcnJ85jEumJXPWuMFEhAbWVPRKdcYJbSI/BJZYz5PxJJU2hdY2gIJjtp/uy6Au/PtMBP/NP6O8qyrEEG+EQc/+ms3mNwAYzNeOMV9/+dVe09nRnRx3go1tT93GcC6wKCqMwckRxA8IJ6RRYD2eh1K+NHwyLPB9rb/PkoiIrASGd7Lrl8aY161jfgm0AM+1va2T4w2dd0U+0d84InI9cD3AyJEjexD1V5KJItTt+ykDlI+44cz6cGLDQ7/2SyUAHdocvrbvmN++446SzvZ1ToDIsBASYsIJc2lPehW8fJZEjDHnd7VfRK4BvgWcZ0z7Z8JCILXDYSlA22LCJ9re2bUfBx4HyMrKOmGy6cpT163tzduUUqpfseUjktXT6k5goTGmrsOuN4CrRCRCRNKBscAGYCMwVkTSRSQcT+P7G/6OWyml1NfZ1SbyCBABvG91Z/zMGHOjMWa7iCzF02DeAvy7MaYVQERuAVbg6eL7lDFmuz2hK6WUaiPm2NbFIJOVlWWys7PtDkMppQKGiGwyxmR151ht8VNKKdVrmkSUUkr1miYRpZRSvaZJRCmlVK9pElFKKdVrQd87S0TKgP29fPtg4LAXwwk0ev96/3r//dMoY8yQ7hwY9EmkL0Qku7vd3IKR3r/ev95//73/7tLqLKWUUr2mSUQppVSvaRLp2uN2B2Azvf/+Te9fnZS2iSillOo1LYkopZTqNU0inRCR+SKyS0RyRcSe9VV9QERSRWSNiOSIyHYRudXaniAi74vIHutrvLVdROQh6/uwRURO63Cua6zj91hrwwQMEQkRkS9E5C3rdbqIrLfuZYm13ADWkgRLrPtfLyJpHc5xt7V9l4hcaM+d9JyIDBKRl0Vkp/V7cEZ/+vmLyG3W7/42EXlBRCL708/fJ4wx+ujwwDPV/F5gNBAOfAlMsDsuL91bEnCa9TwW2A1MAO4H7rK23wX8wXp+EfAunoX6ZgHrre0JwD7ra7z1PN7u++vB9+E/gOeBt6zXS4GrrOd/A26ynt8M/M16fhWwxHo+wfq9iADSrd+XELvvq5v3/g/gx9bzcGBQf/n541lqOw+I6vBz/0F/+vn74qElkePNBHKNMfuMMU3Ai8AlNsfkFcaYImPM59bzGiAHzx/WJXj+uWB9vdR6fgnwrPH4DBgkIknAhcD7xpgKY0wl8D4w34+30msikgJ8E3jCei3AucDL1iHH3n/b9+Vl4Dzr+EuAF40xjcaYPCAXz++No4lIHHAW8CSAMabJGHOEfvTzx7OGUpSIhALRQBH95OfvK5pEjpcMFHR4XWhtCypW0fxUYD0wzBhTBJ5EAwy1DjvR9yKQv0cPAncAbut1InDEGNNive54L+33ae2vso4P1PsfDZQBT1vVeU+IyAD6yc/fGHMQ+F/gAJ7kUQVsov/8/H1Ck8jxpJNtQdWFTURigFeAnxljqrs6tJNtpovtjiYi3wJKjTGbOm7u5FBzkn0Bef94PoWfBvzVGHMqUIun+upEgur+rbaeS/BUQY0ABgALOjk0WH/+PqFJ5HiFQGqH1ynAIZti8ToRCcOTQJ4zxrxqbS6xqimwvpZa20/0vQjU79EcYKGI5OOppjwXT8lkkFW9AV+/l/b7tPYPBCoI3PsvBAqNMeut1y/jSSr95ed/PpBnjCkzxjQDrwKz6T8/f5/QJHK8jcBYq8dGOJ4GtTdsjskrrPrcJ4EcY8wfO+x6A2jrYXMN8HqH7d+3eunMAqqs6o4VwDdEJN76dPcNa5ujGWPuNsakGGPS8PxcVxtjrgbWAJdbhx17/23fl8ut4421/Sqr9046MBbY4Kfb6DVjTDFQICLjrU3nATvoJz9/PNVYs0Qk2vpbaLv/fvHz9xm7W/ad+MDTK2U3nl4Xv7Q7Hi/e11w8xe4twGbrcRGeet5VwB7ra4J1vACPWt+HrUBWh3P9EE+DYi5wrd331ovvxTy+6p01Gs8/gVzgJSDC2h5pvc619o/u8P5fWt+XXcACu++nB/c9Dci2fgdew9O7qt/8/IF7gZ3ANuCfeHpY9Zufvy8eOmJdKUGVK0wAAAJ7SURBVKVUr2l1llJKqV7TJKKUUqrXNIkopZTqNU0iSimlek2TiFJKqV7TJKJUN4hIq4hs7vDocnZnEblRRL7vhevmi8jgvp5HKV/RLr5KdYOIHDXGxNhw3Xw84zMO+/vaSnWHlkSU6gOrpPAHEdlgPcZY2+8Rkdut5z8VkR3WmhwvWtsSROQ1a9tnIjLF2p4oIu9ZEyQ+Rod5mkTk36xrbBaRx8SzLkqIiDxjrY+xVURus+HboPoxTSJKdU/UMdVZV3bYV22MmQk8gmcurmPdBZxqjJkC3Ghtuxf4wtr2n8Cz1vbfAh8ZzwSJbwAjAUQkE7gSmGOMmQa0AlfjGYGebIyZZIyZDDztxXtW6qRCT36IUgqot/55d+aFDl//1Mn+LcBzIvIanqlGwDMFzbcBjDGrrRLIQDzrfVxmbX9bRCqt488DpgMbPdM+EYVnosQ3gdEi8jDwNvBe729RqZ7TkohSfWdO8LzNN/HMQTUd2GTNCNvVdOKdnUOAfxhjplmP8caYe4xnUaipwAfAv2MttqWUv2gSUarvruzw9dOOO0TEBaQaY9bgWQxrEBADrMNTHYWIzAMOG8/aLh23L8AzQSJ4Jka8XESGWvsSRGSU1XPLZYx5Bfg1nqndlfIbrc5SqnuiRGRzh9fLjTFt3XwjRGQ9ng9l3znmfSHAv6yqKgH+ZIw5IiL34FlhcAtQx1dTjt8LvCAinwNr8UxfjjFmh4j8CnjPSkzNeEoe9dZ52j4Q3u29W1bq5LSLr1J9oF1wVX+n1VlKKaV6TUsiSimlek1LIkoppXpNk4hSSqle0ySilFKq1zSJKKWU6jVNIkoppXpNk4hSSqle+/9hr56M9KVj3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label = 'avg')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label = 'min')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label = 'max')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Rewards')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Rendering Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "done = False\n",
    "state = env.reset()\n",
    "while not done:\n",
    "    action = np.argmax(q_table[get_discrete_state(state)])\n",
    "    next_state, _, done, _ = env.step(action)\n",
    "    state = next_state\n",
    "    env.render()\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python(spinningup)",
   "language": "python",
   "name": "spinningup"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
